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Expert  Elicitation 


“a  formal  process  by  which  expert  judgment  is  obtained  to 
quantify  or  probabilistically  encode  uncertainty  about  some 
uncertain  quantity,  relationship,  parameter,  or  event  of 

decision  relevance” 


USEPA,  Science  Advisory  Board,  2009 
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Some  Applications  of  Expert  Elicitation 

*  Regulatory  settings 

♦  US  Environmental  Protection  Agency 

♦  Nuclear  Regulatory  Commission 

♦  US  Department  of  Agriculture 

•:  Non-Regulatory  settings 

♦  Department  of  Defense 

♦  US  Army  Corps  of  Engineers  (Civil  Works) 

♦  National  Aeronautics  and  Space  Administration 
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Application  of  Expert  Elicitation  to  Emerging 

Contaminants 


Over  -the-  horizon 


EC  News 


Possible  DoD  impacts 


v^atch 


Phase  I 
Assessment 


action 


Probable  high  DoD  impacts 


HI 


Phase  II 
Assessment 


Review  literature, 
periodicals,  regulatory 
communications,  etc. 


Monitor  events;  Conduct 
Phase  I  qualitative 

impact  assessment 


Conduct  Phase  II 
quantitative  impact 
assessment;  develop  & 
rank  RMOs 


risk 

^anageme^  ^  Risk  Management  Options  (RMOs)  to  ECGC 

I 

Approved  RMOs  become  Risk  Management  Actions  (RMAs^ 
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Subject  Matter  Expert  Elicitation  in  the 
Phase  I  Impact  Assessment  Process 


Data  Collection 


PH  ASE  I  IMPACT  ASSESSMENT  FOR 
SULFUR  HEXAFLUORIDE  (SF#) 
{CAS  No.  2551-62-4) 


January,  200S 


Distrib  nirou  Statement  D.  Distribution  authorized  to  the  Department  of  Defence  and  U.S.  DcO 
contractors  only  (Administrative  or  Operational  Use)  (14  February'  2008).  Other  requests  for  this 
document  shall  be  referred  to  the  Office  of  die  Deputy' Undersecretary  of  Defense  (I  &  E). 
PtrecTor  of  EmeriiuE  Ccutanrbants.  Ivjt.  Paul  Yaroccdah  a  osd.ahl) 


+ 

Regulatory  Analysis 

_ + _ 

Subject  Matter  Expert 

Input 


Phase  I  Impact 
Assessment 
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Identification  of  Subject  Matter  Experts 
For  Phase  I  Impact  Assessments 

Internal 


♦  Government  employee  experts 

♦  Experts  under  direct  government  contract 


External 

♦  Industrial/Commercial  firms 
and  consultants 

♦  Commercial  manufactures 
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Functional  Areas  for  Impact  Assessment 


Acquisitions  /  Research  Development 
Testing  and  Evaluation 


Production,  Operation,  Maintenance, 
and  Disposal  of  Assets 


Cleanup/Remediation 


Environmental  Safety  and  Health 


Training  and  Readiness 
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Sub-Functional  Area  Assessment 


•  01  -  Infrastructure 

Improvements 

•  02  -  Production  and 

Maintenance 

Operations 

•  03  -  Analytical  Testing 

and  Monitoring  (env) 

•  04  -  Analytical  Testing 

and  Monitoring 
(materiel) 

•  05  -  Material  Handling, 

Storage,  and 
T  ransport 

•  06  -  Waste  Handling, 

Storage, 

Transport,  and 
Disposal 

•  07  -  Personal  Protective 

Equipment 


•  RT1  -  System 

Specific 

•  RT2  -  Training 

Activity 

•  RT3  -  Location 

Specific 


El  -  Human  Health 
E2  -  Occupational  Health 
E3  -  Safety 

E4  -  Environmental  Health 
E5  -  Community,  Public  or 
Worker  Relations 
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Severity  of  Adverse  Impact  |_| 

ES&H  POMD  of  Assets 

Readiness  &  Training  X  Cleanup 


R1  -  Material  Availability 
R2  -  Material  Qualifications 
R3  -  Industrial  Base 

Equipment  Suppliers 
R4  -  Cost  and  Schedule 
R5  -  Materials  Laboratory 
Scale  Activities 


•  Cl  -  New  Site 

Identification 

•  C2  -  Remedial 

Technologies 

•  C3  -  Existing  Sites 

•  C4  -  Cost  to  Complete 

•  C5  -  Property  Transfer 

and  Re-Use 


Acquisition/RDT&E 
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Types  of  Subject  Matter  Experts  (SMEs) 


Functional  Area 

Total 

SME 

Non-SME 

Acquisition  /  Research, 
Development,  Testing,  and 
Evaluation 

30 

12 

18 

Clean  Up 

30 

8 

22 

Environment,  Safety,  and 
Health 

30 

21 

9 

Production,  Operations, 
Maintenance,  and 

Disposal  of  Assets 

30 

11 

19 

Readiness  and  Training 

30 

4 

26 

Data  from:  Cadmium  and  Compounds,  November  2009 
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Expert  Elicitation  Process  (Probability) 


•t  Probability  of  Adverse  Impact  (1-5) 

♦  Definite  >  80%  Will  definitely  be  affected 

♦  Probable  51-80%  Will  probably  be  affected 

♦  Possible  31-50%  Will  possibly  be  affected 

♦  Plausible  1 1  -30%  Slight  possibility  of  affect 


♦  Improbable  <  10%  Unlikely  to  have  affect  or  no 
affect 
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Expert  Elicitation  Process  (Severity) 


•:  Severity  of  Adverse  Impact  (1-5) 


♦  Unacceptable  increases  in  the  number,  type  and  degree  of 
human  health  hazards  for  non  DoD  employees  or  offsite 
populations 


♦  Significant  increases  the  number  type  and  degree  of  human 
health  hazards  for  non  DoD  employees  or  offsite  populations 

♦  Moderate  increases  the  number  type  and  degree  of  human 
health  hazards  for  non  DoD  employees  or  offsite 
populations. 

♦  Limited  increases  the  number  type  and  degree  of  human 
health  hazards  for  non  DoD  employees  or  offsite  populations 

♦  No  non  DoD  employees  or  offsite  populations  are 
considered  at  elevated  risk  as  a  result  of  DoD  activities. 
Hazards  are  inconsequential 
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Expert  Elicitation  Process  (Confidence) 


•:  Confidence  in  Probability  and  Severity  (1-5) 

♦  Certain  -  Supported  by  facts  and  experience 

♦  Somewhat  certain  -  Supported  by  some  facts, 
experience  or  anecdotal  information 

♦  Not  certain  -  Intuitive  answer  based  on  similar 
situations 

♦  Somewhat  unsure  -  Intuitive  answer  but  no  basis 

♦  Uncertain  ' guesstimate ‘  -  No  facts,  experience  or 
intuition 
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Emerging  Contaminant  Assessment  System 

(EC  AS) 
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Example  from  Expert  Elicitation 


•:  Acquisition  /  Research, 
Development,  Testing, 
and  Evaluation 

•:  Question  2  -  Material 
Qualifications: 

♦  What  is  the  risk  that 
increasing  international 
regulatory  restrictions 
[e.g.,  REACH  or  RoHS] 
for  cadmium  and 
compounds  will  increase 
testing,  specification,  and 
evaluation  requirements? 


Data  from  Cadmium  and  Compounds,  Oct  2009 
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Expert  Elicitation  Data  Summary 
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PI  D  I  t> - 1 - L  \  \  ] - t* 

Combined  Functional  Areas 


n  f  t  f  iy  r  n  f  t  r  m 

E-ESH 

El  -  Human  Health 

E2  -  Occupational  Health  (civilian  and  uniformed) 

E3  -  Safety 

E4  •  Environmental  Health 

E5  -  Community  Public  or  Worker  Relations 

RT . T&R 

RT1  -Training  Facilities  (System  Specific) 

RT2  -  Training  Activities  (Activity  Specific) 

RT3  -  Military  Readiness  (Location  Specific) 

R  -  Acq-RDT&E 
R1  -  Matenal  Availability 
R2  -  Matenal  Qualifications 
R3  -  Industnal  Base  and  Equipment  Suppliers 
R4  -  Cost  and  Schedule 
R5  -  Matenals  Laboratory  Scale  Activities 
O-POMD 

01  -  Infrastructure  Improvements 
02  -  Production  and  Maintenance  Operations 
03  -  Analytical  Testing  and  Momtonng 
04  -  Analytical  Testing  and  Momtonng 
05  -  Matenal  Handling  Storage  and  Transport 
(HS&T) 

06  -  Waste  Handling  Storage  Transport  and 
Disposal  iHST&D) 

07  -  Personal  Protective  Equipment  (PPE) 

C  -  Cleanup 

Cl  -  New  Site  Identification 
C2  -  Remedial  Technologies 
C3  -  Existing  Sites 
C4  -  Cost  to  Complete 
C5  -  Property  Transfer  and  Re-Use 


slZH 


T6A 


AccH 


£p\/prifv 

POMP  A  Cleanup 


Data  from  Cerium  and  Compounds,  July  2009 
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Phase  I  Impact  Assessments  Completed 


S  Tungsten 

s  Tungsten  alloy 

s  Tetrachloroethylene(PCE) 

■s  Dioxins 

s  1 ,4-Dioxane 

s  Perfluorooctyl  sulfonate 
(PFOS) 

■s  Di-nitrotoluenes  (DNT) 

■s  Lead 
✓  Nickel 

v'  Hexavalent  Chromium 
s  Cerium 

■s  Trichloroethylene  (TCE) 


S  Sulfur  Hexafluoride  (SF6) 

■s  Polybrominated  diphenyl 
ethers  (PBDEs) 

■s  1 ,2,3-trichloropropane  (TCP) 

■s  N-nitrosodimethylamine 
(NDMA) 

•s  Dichlorobenzenes 
^  Beryllium 
^  Naphthalene 

■s  Perfluorooctanoic  Acid  (PFOA) 
^  RDX 

Cadmium  (report  in  preparation) 

^  Antimony  (report  in  preparation) 
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Next  Steps 

4  Apply  a  continuous  process 
improvement 

4  Integrate  confidence  measurements 

4  Continue  outreach  to  improve  breadth 
and  depth  of  pool  of  subject  matter 
experts 

•:  Independent  review  of  expert 
elicitation  process 
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Future  Phase  I  Impact  Assessment 
Subject  Matter  Expert  Meetings 


CHEMICAL 

CAS  NUMBER 

PROJECTED  MEETING  DATE 

Dinitrotoluene  (DNT) 

25321-14-6 

July  2010 

Nanomaterials 

— 

October  2010 

Phthalate  Esters 

— 

January  2010 

1 ,4-dioxane 

123-91-1 

Future  reassessment  TBD 

Antimony  and  Compounds 

7440-36-0 

Future  assessment  TBD 

Cobalt  and  Compounds 

7440-48-4 

Future  reassessment  TBD 

Nickel 

7440-02-0 

Future  reassessment  TBD 

Tungstate 

7440-33-7 

Future  reassessment  TBD 
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